It's a port on Hurtigruten, and a quick visit to the fort can just be managed on the southbound voyage.
Vardø is the endpoint for E75.
Vardø has a small airport IATA: VAW, witch has flights to Alta and Kirkenes.